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FOREWORD 



Educational development Is emerging as a new discipline. As yet, 

Its principal features are only generally understood, but Its significance 
as a tool for educational reform cannot bo seriously questioned. This 
case $tu<ty of the development of a specific educational product by the 
Far West Laboratory for Educational Research and Development should prove 
helpful to those who wish to gain a better understanding of this new 
discipline. 

The directors of regional educational research and development lab- 
oratories are being constantly bombarded with questions such as the 
following: 

1. What are regional educational laboratories doing? 

2. Exactly what Is "educational development?" 

3. What Is the relationship between educational research and 
development? 

4. Are special laboratories needed to perform educational develop- 
ment? Could It not be done better by a university- -state 
departments of educat1on--loca1 teaching faculties? 

5. What skills are needed by those who might consider a career In 
educational development? 

6. When is the work of an educational development completed? How 
locoes It take to develop a given product? 

7. Hew much does It cost to develop a given educational product? 
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AN EDUCATIONAL DEVELOPMENT CASE STUDY: 

AN ELEMENTARY SCIENCE INFORMATION UNIT 

SUMMARY 

The nurrber of new processes and products available to schools Is 
Increasing at a rapid rate. They are relatively complex and difficult to 
understand. On the surface they are no more appealing than the great 
quantity of material already available. We must ask ourselves: 

Do teachers and administrators have enough time to find, gather, 

and process Information about the new developments? 

Do researchers and developers adequately disseminate Information 

that Is readable and understandable? 

The Far West Laboratory believes that the answer to these questions 
Is "no." As a result, the Laboratory has developed the E ^mentary Science 
Information Unit , lhe unit describes six new, relatively well developed 
programs suitable for science Instruction In elementary schools. The 
Information unit decreases the work load of the school personnel respon- 
sible for reviewing these developments for possible adoption, tt assures 
that Information about them Is presented In a clear, effective format. 

A separate report published by the Laboratory, "The Final Report of 
the Elementary Science Unit," describes the Information unit and Its use 
In detail. The current study Is Intended to be a record o* the development 
effort behind the product. 

This case stu<1y focuses on five major steps used to conceptualize, 
develop, and test the product: 

In the first phase, Conceptualization and Planning , the goals of the 




unit were outlined. Brief ly * the goal was to close the gap between the 
schools and the emerging field of educational research and development by 

111 



increasing the number of well developed options known to school decision- 
makers and by conveying knowledge about the alternatives to enable schools 
to accept or reject them rationally. The model developed was a multi- 
level, multi-media, mailable product describing alternatives In an objective 
manner. A non-directive Instructional approach was adopted. A prototype of 
the unit was developed and submitted for "expert" evaluation. A pattern for 
field testing and evaluation was also devised, as well as a strategy for 
Increasing the knowledge base behind diffusion efforts such as this one. 

In the second phase, Preliminary Product Development and Testing , the 
first complete form of the Information unit was developed. A strategy for 
selecting the science programs to be Included was Implemented, Information 
was gathered and the specific elements of the product were written and 
produced. A field test Involving 19 target audience subjects was conducted. 
The results Indicated that there was a real need for the product and that 
the format was largely satisfactory, although a number of small modifications 
were recommended. Simultaneously, the subjects provided data on the relation- 
ship of "job function" to "search-set." Among other outcomes of this research 
was the conclusion that formal Job classification (e.g., superintendent vs. 
teacher) has little to do with the type and nature of Information required. 

In the third stage, the Development and Field Testing of the Main Form , 
a completely revised unit was developed using the modifications recomnended 
at the previous stage. One Important change had to do with the media used. 
Field testing with a randomly selected sample of 19 schools was conducted. 

One hundred eighty-one target users were Involved. Results Indicated that 
the unit had satisfactorily passed all of the standards set for Its objectives 
except In one area--kncw1edge retention. Reconmcndatlons were made to 
Improve product performance In this area. 



In the fourth stage, the Development and Field Testing of the 



Operational Form , the product was revised and used in a large number of sites 
across the country. This time the subjects were free to use the product in 
any way they wanted {as opposed to the controlled testing of the previous 
stage). Twenty- four schools were involved. Results were highly satisfactory 
and the product was recommended for release. 

In the fifth stage, Development of the Final Product and Dissemination , 
considerable difficulties were encountered in securing a commercial dis- 
tributor, resulting In the need for a relatively large amount of extra money 
to develop the final form. Dissemination of this form has now begun although 
no information on the number of sales is currently available except to note 
that 130 pre-publication sales were made at $75 each. 

Elsewhere in this document, the background of the program which 
produced the product is described, legal constraints are outlined, and a 
"system development strategy" is reported. 

Costs for the effort total approximately $222,000, excluding legal fees 
and the cost of the copies of the final form (to be recovered from sales). 

A breakdown of the budget data shows the cost of each stage as follows: 



Conceptualization and Planning $24,000 
Preliminary Development and Testing 45,000 
Main Form Development and Testing 46,200 
Operational Form Development and Testing 47,800 
Final Form 59,000 

Total $222,000 



Of the total, approximately 50X ($110,000) was spent for personnel 
(salaries, wages, and benefits). Costs for materials and subcontracts 
related to the production of audiovisual materials (exclusive of personnel 
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costs) were approximately 14% ($31,000). The remainder (36%) went to all 
other costs— Laboratory management support, indirect costs (heat, light, 
communications, etc.), and other direct costs. 

Another way of breaking down this total figure shows that a total of 
$117,000 (53%) went for actual development of the forms (information 
collecting, writing, typing, shooting photographies, revision, etc.), 

$38,000 (17%) went into field testing (travel, payment to participants, data 
analysis, etc.), $9,000 (4%) went into reporting (to Laboratory management, 
U.S.O.E., and the educational community in general), and the remainder of 
$58,000 (26%) went into management, recruiting, training, coordination, etc. 

The case study also points to other payoffs this effort has had by: 

(1) allowing the program to develop similar products at less cost ( a com- 
parable information unit now under development is running at about 50% of 
the cost of the science unit) and (2) permitting an entire reconceptualization 
of the system so that it can meet more of its intended objectives as well as 
serve a larger scope of effort. The second payoff will result in a new 
product development phase, which is now under design; the planning document 
for this new design is unique— a complete design specification of an 
educational development. 



vi 
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INTRODUCTION TO THE PROBLEM 



In 1965 Congress enacted the Elementary and Secondary Education Act, 
Public Law 89-10. Under Title VI of this act the Office of Education 
established a program of regional laboratories. The model for the 
laboratories left room for a variety of Interpretations (see the "Hearings 
Before the Special Subcommittee on Education on the Study of USOE, 89th 
Congress"!) but it was clear that a new means was proposed to develop 
programs for the schools. Instead of relying on the "dissertation model" 
of research for new Ideas and products, the laboratories were seen as 
"permanent institutions devoted to the discovery and dissemination of 

O 

practical knowledge..." 

After the Initial planning during 1965, the Far West Laboratory for 
Educational Research and Development was established In 1966 as a joint 
agreement between a number of California and Nevada educational agencies. 3 



lu.S. .Congress , House Committee on Education and Labor, Subcommittee 
on Education, Hearings, Study of the United States Office of Education (89th 
Congress. 1967, four parts). Also see Stephen K. Bailey, "Emergence of the 
Laboratory Program," Journal of Research and Development in Education, III 
(1970), 11-13. 

o 

Bailey, 0 £. cit . , p. 6. 

Original signatories to the Joint Powers Agreement of 1966 were: The 
Regents of the University of California, the California Board of Education, 
the Trustees of the California State Colleges, the County Superintendent 
of Schools of the County of Monterey, the Board of Education of the San 
Francisco Unified School District, the Regents of the University of Nevada, 
and the Nevada State Board of Education. In 1969 the Board of Regents, 
University of Utah and the Utah State Board of Education were added to the 
signatories. 
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In Its initial definition of focus the Laboratory staff identified six 

alternative areas for possible program development. Using a number of 

criteria, the six areas were narrowed to two major programs. The criteria 

used were: "importance, focus, breadth of tasks, payoff, feedback, 

compatibility with resources, organizational involvement, fundamental 

problem, potential duplication of effort, funding feasibility, breadth of 

application, political feasibility, balance of tasks, and overall risk." 

The "Communication Program" was one of those selected for initial 

development. The other program was "Teacher Education. 

The Purpose of the Communi cation Program was: 

To conduct those research, developmental and operational tasks 
that will bring into existence effective use of information 
about options available to school personnel as they make deci- 
sions in the organizational operations of schools.* 

This goal was selected in an effort to close the gap between the schools 

and the emerging field of educational research and development. There was 

ample evidence that research had not found its way Into established practice 

and it was important that the pattern not be repeated in the newly undertaken 

development effort. 

Initial conceptualization of the components of the Communi cation 
Program took place between 1966 and 1967. The components identified were to 
focus on three efforts: 



^Far West Laboratory for Educational Research and Development, Program 
Plans (Berkeley: Far West Laboratory for Educational Research and Develop- 
ment, March, 1967), pp. 7-8. 

5 IMd, o. 42. 
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Component 1: The development and refinement of media products to 

create awareness, realistic expectations, positive attitudes 
and motivation, and a supportive climate relating to research 
and development and rational educational planning;** 

Component 2: The development and establishment of an effective 

Information system, specifically designed for school 
personnel and limited to educational research and development 
information; 

Component 3: The identification, development, and dissemination 

of organizational arrangements and personnel training programs 
that will permit school personnel to use research and 
development information effectively. ^ 

This document Is a case study of the first product of Component 2-- 
the information system. The product is the Elementary Science Information 
Unit . The case study is Intended to provide sufficient detail to enable 
the reader to understand the complexity and costs involved in the empirically 
based development of an education product. 

The strategy used to Identify, create, test, revise, retest, and 
market the product is patterned after industrial and military models. The 
Laboratory staff identified five major stages through which the product 
had to pass. The stages were further subdivided Into more detailed phases. 
The bulk of this document describes these phases In terms of the development 
of the Elementary Science Information Unit . 

^The goals of the component were eventually changed. 

?Far West Laboratory for Educational Research and Development. Basic 
P rogram Plans (Berkeley: Far West Laboratory for Educational Research and 

Development, 1968). 
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STAGES OF DEVELOPMENT 

Stage 1 : Conceptualization and Planning 

Purpose of This Stage . As defined by the overall developmental 

strategy of the Laboratory, this stage Included: 

need definition, a thorough review of the research literature and 
practices that seem to be relevant to the particular needs and 
problems on which the program or component Is focused, a detailed 
statement of objectives to be achieved through the use of the 
product and preparation of Initial specifications of the product. 8 

Steps of This Stage . For the purposes of this paper the work of 

stage 1 will be divided Into eight subsections: Identifying the research 

base, describing the requirements, goals and target audience, developing 

the prototype and model, selection of content, production and scheduling, 

personnel, time and costs. 

Identifying the research base . Literature searches and professional 
judgement led the staff to conclude that an extensive research base did 
not exist for the development of a product in the proposed area. While 
research had been conducted on the general variables that Influence the 
adoption-decision process, 9 little of it suggested a specific developmental 
strategy. The most relevant work was from rural sociology. Rogers, for 
example, was instrumental In pointing to the importance of a successful 

O 

far West Laboratory for Educational Research and Development, 
Contractor's Request . (Berkeley: Far West Laboratory for Educational 
Research and Development, 1970). 

9 For a good current summary see R. G. Havelock, Planning for 
Innovation (Ann Arbor: University of Michigan, Institute for Social 

Research, 1969). 
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product being divisible, simple (as opposed to complex), easy to understand 
(communicable), compatible with existing procedures, and clearly 
advantageous over alternatives.^ 

From Guba^the staff adopted a specific product development 
strategy of telling and showing rather than "helping, Involving, training, 
and Intervening," "Telling and showing" seemed most likely to produce a 
cost-effective approach to the goals of the component. In other 
words, it was believed that In order to reach the largest target audience 
most effectively, any idea or practice had to be formed into a tangible, 
operational product that could be distributed easily and Inexpensively. 

While other strategies involving direct services to schools might be 
more powerful in the Isolated case, it was judged that it would be most 
cost-effective to create a mailable product describing new Innovations in 
a decision-making framework. The staff recognized, of course, that if 
this approach were not well conceived and tested, effectiveness would 
be low since the approach was not closely coupled with the schools' 
current practices. 

Describing the requirements, goals, and target audience . In a series 
of staff papers the goals and the target audience were Identified. Briefly, 

f 

the goals were identified as: 

^E. M. r.ogers, D iffusion of Innovations . (New York: The Free Press 
or Glencoe, Inc., 1967). 

^E. G. Guba, "Development, Diffusion and Evaluation," Kn owledge 
Production and Utilization , eds. T. L. Ei del 1 and Joanne M. Kltchel , 
University Council on Educational Administration and the Center for 
Advanced Study of Educational Administration (Eugene: University of 
Oregon, 1968). 




(1) Increasing the number of well researched and developed program 
options known to school decision makers; 

(2) Conveying specific knowledge of these programs sufficient for: 

(3) Educational decision-makers to select from among the 
alternatives the one best suited to their goals or reject them 
all if none were satisfactory. 

In addition, secondary goals were developed: 

(4) The target audience must like the product developed; 

(5) believe it serves their needs, and 

(6) prefer it to all other equal or less costly means of 
achieving the same objectives. 

The target audience for the proposed product were those members of 
a local educational unit (school building or district) responsible for 
curriculum decision-making. It was assumed that almost all of a district' 
personnel might be involved at some poirit--assistant teachers, teachers, 
department heads, vice-principals, principals, supervisors, coordinators, 
consultants, deputy/associate/assistant superintendents, superintendents, 
community advisory groups, and school boards. However, it was assumed 
that the principals, supervisors, coordinators, consultants and teachers 
(when the latter served in an administrative capacity) were the key 
members of the target audience. 

In a detailed analysis of these potential members of the target 
audience^ it was assumed that not all of these decision-makers had the 
same information needs. Differences existed both in terms of the type 
and detail of information required by the target audience. Thus it was 
clear that whatever product eventually emerged had to be flexible both In 

12paul D. Hood, "The Integrated Information Unit and Its Possible 
Utilization by School Personnel" (unpublished paper, Far West Laboratory 
for Educational Research and Development, 1967). 
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terms of type of information (content, objectives, training 
requirements, price, etc.) and detail (from short summaries to 
detailed technical reports). This decision seemed compatible with the 
research work of Rogers mentioned previously. 

Developi n g the prototype and model . During the first phase of 
the developmental cycle the Laboratory commissioned the Lockheed Missiles 
and Space Company to conduct a detailed survey of the Information needs 
of the target area of the Laboratory. The report was completed in November 
of 1966. Though the report had many purposes and other conclusions, the 
findings relevant to the present case study Included the$-_ v.ostracted Items: 

(1) There was general lament from all levels of education about 
the lack of communication in the field and about the number of 
independent efforts that were proceeding without awareness of 
each other. 

(2) There was a lack of significant research Information to 
disseminate. 

(3) Research findings needed to be translated Into a form which Is 
understandable by the school constituency. 

(4) There were a large number of media channels that could be 
utilized; most were In the sphere of either the mass media 
or the highly technical area of computer science. 13 

The results of the survey Included the definition of a number of specific 
vehicles that might be used to effect the needed communications link. These 
Included: the development of a "data book" that would be available to all 

school personnel, a laboratory publication service, a program "alert" service 

1 3 

Lockheed Missiles & Space Company, Communication and Utilization 
Study (Sunnyvale, Calif.: Lockheed Missiles & Space Company, 1966). 
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utilizing existing mass media channels, development of a systems planning 
guide (a handbook), and the operation of a field demonstration unit. The 
development of a large scale computer-based Information system was also 
suggested. 

Eventually all of these Ideas were rejected though the study served 
to confirm the need for an information system that would enable school 
personnel to know of educational developments and select among them. The 
reasons that the alternatives were rejected Included these: While highly 
visible, It was believed the "data book" would not be well focused and 
might degenerate into "the stuff" that tends to pile up on desks and In 
files; it was believed that the product would have to be more unique. 

The publication service was rejected because other agencies already provided 
that service. The program "alert" was tentatively selected as a means for 
Implementing Component 1 ' s objectives (see above); however, when a series of 
educational television programs developed as an "alert" reached only 5% of 
the potential target audience It was dropped. The system planning guide 
was rejected as not being enough on target for the purposes of the product, 
the field demonstration unit violated the "telling-showing" product 
development constraint and was thought to be too costly for the small target 
audience that would be reached. The use of a computer-based Information 
system was rejected on the grounds that not enough schools now had such facl 11 tie 
(and were unlikely to get them In the near future) and the Interconnection 
costs were too expensive at present. 



14 Ib1d. 
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The eventual model that was conceived for the product was one 
Invented by the staff of the Laboratory. It consisted of a multi-level 
package of multi-media materials suitable for use by school personnel 
as they considered alternative ways to achieve their educational goals 
and objectives. An assumption was made that providing Information about 
several feasible alternatives In thr same product would assist the 
rationality of the decision process It was also assumed that If all or 
most of the critical information (pro and con) was made available In a 
useful format there would be a better chance that the right choice would 
be made for the particular situation. 

In the March 19 Laboratory Activities a schematic of the proposed 
product was Introduced. (See figure 1.) 

The name assigned to the product was the Integrated Information Unit. 
The term "Integrated," originally Included to suggest the multi-media, 
multi-level format, was eventually dropped for simplicity. 

As shown in the schematic, the model had four levels of information-- 
an Initial overview to all the programs, a more detailed audiovisual 
summary of each program, a booklet describing each program In more detail, 
and finally, a set of background references. Film was used at levels one 
and two, print at levels three and four. Each program was treated in a 
parallel way to permit adequate comparison between the alternative 
programs. It was decided that each Information unit would be focused 
on a single set of alternatives in a given area. 



ORGANIZATION OF AN INTEGRATED 
INFORMATION UNIT 






Figure 1 

The Proposed tadel for the Information Unit 
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The instructional pattern behind this mock;! Included 'j series of 
assumptions: 



1. The curriculum-adoption, decision -making practices of schools 
are extraordinarily varied; no single model of how needs are assessed, 
goals and objectives identified, Information about alternatives 
gathered and processed, --and decisions made can be established 

among school organizations. It Is assumed that the processes 
In most cases Involve multiple, recurring encounters with needs, 
goals, and 1nformat1on--rather than a one-dimensional model of the 
processes. 

2. It Is assumed that a great variety of personnel Is 
1nvolved--rang1ng from parents and students to teachers and adminis- 
trators. It Is also believed that the variation In Information 
processing behaviors is as broad within groups as It Is among 
groups, in general, the process Is assumed to involve both group 
and individual acts ranging over a period of time. 

3. It Is assumed that the types of Information needed will 
vary greatly from one Individual to another and from one group to 
another. 



4. It Is assumed that motivation to want and need the type of 
information provided by the unit must exist before coming to the unit; 
the unit cannot Induce review and rational consideration of new 
alternatives leading to adoption unless the user Is Inclined in this 
direction. 

5. It Is also assumed that a mailable package of Information 
about new curriculum projects Is a necessary but not sufficient 
stimulus for rational decision-making; other types of materials and 
contacts must also te present to lead to the terminal behaviors of 
adoption, adaptation or rejection. 15 

Most of these assumptions were untested and research about them was 
lacking. As a result, the choice of strategies was either to make h 
considerable Initial effort to find evlden.e supporting or rejecting 
these assumptions or to begin development and attempt to establish 
supporting research as we went along. The latter course of action 
was the one adopted. In this specific case, the result In design 
Included these instructional characteristics: 



15 C. L. Hutchins, A Final Report on the Elementary Science Information 
Unit {8erke1ey: Far West laboratory for Educational Research and bevelop- 

ment, 1970). 

o 
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1. Except for the initial instructions, the sequence for using 
the unit was not linear or discursive; entry to the body o. e the con- 
t?nt was by multiple paths suiting the variety of styles used by 
Individuals and organizations. Hence, the need for each major part 
to be relatively independent of the others. Redundancy exists where 
the prerequisite Information is necessary to understand particular 
cognitive elements. 

2. The Information must te available In multiple forms suitable 
for use by Individuals and groups. 

3. The Information must cover a range wide enough to fit 
multiple needs; If an error Is committed It should be on the side of 
comprehensiveness rather than exclusiveness. 

4. Though motivation is presumed, the unit should provide 
sufficient "persuasion" that tne user already Inclined toward the use 
of such an approach will be Induced to believe that the product Is 
well-adapted to tils needs. 

5. The package must be flexible enough to be used as an auxiliary 
to other materials and In a variety of situations Involving Its use 

by "linking agents," peers, and such formal chanr* 1 : as pre-service 
education. i 6 

Alternative means could have been selected. The staff described 
these alternatives: 

The Information could have been brochures, pamphlets, position papers, 
etc. from the developers of the new science programs themselves. 

This approach was rejected because Information about different 
programs wojld r. '. be comparable, l.e., objective. Not all of the 
developments had suitable materials available. 

The Information could have been the curricula themselves. This 
approach was rejected because of the high cost of most of the 
programs (complete sets of materials could easily have run to 
several thousand dollars). 

The approach could have been more of a demonstration project 
requiring the users to come to some national, regional or local 
Information center. This was rejected because of the additional 
costs and Inconveniences required for use and the fewer number 
of users that would have, In all likelihood, used the product. 



16 Ib1d. 
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The direct, demonstration approach was also rejected because 
to some extent curriculum Information centers already exist In 
large school, county, regional, state or national libraries and 
demonstration centers. 

The use of the package could have been more directed or guided; 
the pattern of use could have been "programmed." This alternative 
was rejected because of the absence of Information on how users 
currently make curriculum reviews; It was assumed that different 
groups follow strikingly different patterns of review. In other 
wcrds, the product was designed to fit present practices rather 
than developing a new pattern that would require major changes In 
most users' behavior. 

A more active (rather than passive) approach probably could have 
been designed. Any failure here was more a lack of Inventiveness 
than systematic rejection of alternatives. 

The product could have used a single level of Information approach 
and a single medium. The multi-level, multi-media approach was 
selected to Insure versatility and flexibility In adapting to 
the user's present practices. 17 

The model that evolved from the research and testing cycle has a 
remarkable overall similarity to the one proposed during the conceptual 
planning. However, there have been a number of changes. It Is the Intent 
of this paper to document these changes and to suggest that the Initially 
conceived product, If executed and disseminated Immediately, would have 
failed. Only the many changes that have occurred--smal1 , though some of 
them may be--make the final product a successful one--not Just because 
of Its effects on users, but also because of Its total reflection of a 
system of production, testing, marketing, etc. 

Selection of the content . During the Planning and Conceptualization 
stage, work was conducted to Identify the subject area In which the first 
Information unit would be developed and to select the specific programs 
that would be covered. During the summer and fall of 1967 a contract was 




o 
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let with Davidson Films to conduct a survey of potential subject areas and 
to help to Identify programs for ultimate Inclusion In the Information unit. 
Davidson Films was selected because of the experience Mr. Davidson and 
his staff had with a number of the evolving "new curricula" and because of 
his knowledge of their suitability to the Information unit approach which 
had been selected. The contract al>o focused on Identifying the media 
that should be used In developing the information unit. The survey 
Included personal Interviews and visits with leading media agencies (DA'/I, 
Eastman Kodak, etc.) leading educational developers (EDC, NSF, the High 
School Geography Project, etc.) and with agencies and groups Interested 
In Information and dissemination (EPIE, EPIC, IDEA, etc.) The conclusions 
of the survey strongly supported the multi -media approach that had been 
selected.^® In consultation with the Laboratory, Davidson also recommended 
that the Initial package should be developed In the field of elementary 
science. It was clear from the outset that because of extensive NSF 
funding, the science and math areas were likely to be the best areas for 
Investigation. Science was selected over math primarily because, at least 
In the Far West laboratory's region, science curriculum selection was In 
much greater ferment than math. Elementary science was selected over 
secondary science because It was felt that the secondary programs were older 
and better known, but that a substantial Investment In elementary science 
curriculum development had been made--1n excess of $10,000,000 at that 
t1me--with few of the products yet being used. 

1 ft 

J. M. Davidson, "Projects Developing Coirmunl cat Ions Systems for 
Educators" (Sar Francisco: Davidson Films, 1967). 




15 



Three criteria were applied to select programs for the Information 
unit. First, the program or curriculum package had to follow the trend 
In elementary science Instruction toward active Involvement of students 
In scientific processes and experience rather than In passive reading or 
listening to accounts of scientific facts, theories, or history. 
Specifically, the program had to provide opportunities for the students 
to manipulate objects and observe functional relationships typical of 
the kind in which scientists engage. To qualify, a program had to either 
include special materials for the students to work with or call for work 
with materials readily available to the typical classroom teacher. Cut 
In either case, such activities could not be appended to a reading chapter 
as "suggestions for further study"; they had to be an Integral part of the 
program. The second criterion was the requirement that the program had 
undergone a research and development cycle comparable to the one developed 
by Far West Laboratory. Specifically, there had to be a systematic attempt 
to field test and operationally confirm that the product worked before its 
release. Furthermore, there had to be some evidence, behavioral or 
otherwise, that the program was meeting Its objectives. In point of fact, 
this was a very difficult criterion to apply since very few curriculum 
developments specify the behavioral objectives required for rigorous 
evaluation. The third criterion was that the materials were, or soon 
would be available for widespread adoption across the country; It seemed 
futile to describe programs that wire unavailable. 

Scheduling . The final step of the Planning and Conceptualitatlon 
phase was to develop a serins of planning documents that could be used to 
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direct production of the Information unlt.^ These documents were focused 
particularly In the area of production and evaluation. PERT schematics 
were drawn up to project a tentative schedule for the development. For 
example, a schematic of the stages Involved In the design of a single film 
In the unit was Included In one of the planning documents. (See figure 2). 
Each step will be fully detailed in the text. 

Personnel . During the Planning and Conceptualization phase the 
project personnel consisted largely of senior staff members. No project 
director had been selected. Planning and conceptualization was the 
responsibility of th oronram director, a Ph.D. in psycholog, with a 
background In Instructional systems development In a military-based 
development agency. Much of the "Inventing" and conceptualization was 
conducted with the aid of the Laboratory Director and the directors of 
other programs. Midway through the planning phase two permanent 
project people were h1red--one at the MA level and one at the BA level. 

One clerical assistant was used about 3/4 time. The backgrounds of 
these people were not In communications or Information dissemination. 

One had background In educational philosophy and the other In social 
studies and the operation of government projects. 



^Paul D. Hood, "The Production of an Integrated Information Unit* 
(Berkeley: Far West laboratory for Educational Research and Development, 

1966), and "The Evaluation of an Integrated Information Unit" (Berkeley: 
Far West Laboratory for Educational Research and Development, 1966). 




Figure 2 
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System Design Outline for Production of a Single 
Audiovisual Component of an Integrated Information Unlt^O 
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Paul D. Hood, "The Production of an Integrated Information Unit." 
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Time and Costs . The conceptualization and planning of the El ementary 
Science Information Unit took place between 1967 and 1968. The cost for 
this stage was approximately $24,000, Including a $3,500 subcontract to 
Davidson Films for a Meld survey. The cost of the Lockheed study Is 
not Included In these figures since that work was used to support the 
design of the entire Communication Program; the Information unit was not 
specifically considered, even though the need assessment features of the 
study were eventually used to support the conceptualization of the unit. 



*■ ’h- percentage of the total costs devoted to personnel and media 
at other stages will be reported separately; a more detailed breakdown will 
also be given. It was not possible to extract the figures for this period. 
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STAGES OF DEVELOPMENT 



Stage 2: Preliminary Product Development and Testing 

Pu rposes of this stage . As currently outlined by the Laboratory, this 
stage Is separated Into two phases: 

Phase 1, preliminary product development , represents all 
the work necessary to create the first form of the product. All 
the Ingenuity and creativity of the staff Is brought to bear 
on the development of what appears to be the most useful pro- 
duct. Frequently non-Laboratory participants and school per- 
sonnel are consulted In the development of the preliminary pro- 
duct! certainly such development Is more than just putting 
together a number of pieces or Ideas that others outside the 
Laboratory have tried out. This phase terminates with the 
decision that the preliminary product Is sufficiently well de- 
fined and developed to merit testing. For some products, this 
preliminary product development may be quite complex, and may 
Include a major portion of the development effort; In other 
Instances the preliminary product may be only a very rough ap- 
proximation of a final product. 

In tha second phase, preliminary field testing , the pro- 
duct Is tested In a preliminary Yie'ld test for Its feasibility 
as an Idea. The evaluation Is most often conducted using re- 
latively small numbers of representatives of the Intended target 
audience who are acquainted with the problems to which the pro- 
duct Is directed. The participants In this preliminary field 
test, or feasibility test, are generally given the opportunity 
to respond freely to questions posed by the staff, as well as 
to draw attention to problems or questions not previously Iden- 
tified by the staff. « 

These phases are expllned under the headings: Selection of the Science 
Program, Gathering the Information, Copyright Constraints, Developing the 
Preliminary Form, the Preliminary Field Test, Conclusions, Personnel, and 
Time and Costs. 



^Far West Laboratory for Educational Research and Development, 
Contractor's Request. 
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Selection of the Science Programs . The general criteria for selecting 
the elementary science programs to be Included in the unit were outlined In 
the description of the planning and conceptualization stage. During 
preliminary development a number of sources were used to Identify a set 
of programs. These Included the Davidson study previously described and 
the International Clearinghouse on Science and Mathematics Curricular 
De velopments . 

The programs selected for Inclusion In the preliminary form of the 
Information unit were: 

Elementary Science Study (ESS) 

Developer: Education Development Center 

Current publisher: McGraw-Hill Book Company, Webster Division. 



Inquiry Development Program (IDP) 

Developer: 0. Richard Suchman 

Current publisher: Science Research Associates 

Minnesota Mathematics and Science Teaching Project (MINNEMAST) 

Developer: Minnesota Mathematics and Science Teaching Project 
Current publisher: MINNEMAST, University of Minnesota 

Science Curriculum Improvement Study (SCIS) 

Developer: Science Curriculum Improvement Study 
Current publisher: Rand McNally and Company 

Sc1encs--A Process Approach (S— APA) 

Developer: The Commission on Science Education of the American 
Association for the Advancement of Science 
Current publisher: Xerox Educational Division 

The science areas covered by these programs were general science, phys- 
ical science, and biological science. Very little astronomy or earth science 
was covered. Mathematics was given equal emphasis In one program (MINNEMAST) 
and was featured prominently In several other programs. 



D. Lockard, Fif th Report of the International Clearinghouse o n 
Science and *tethemat1cs^urr1cular Developments, 196) (College Park: 
University of Maryland, l$6f). 
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The programs were Intended for all elementary school pupils (K-6). 

There was no Indication from the developers that any special segment of the 
student audience (projected high school science majors, slow-1 earners , etc.) 
had been especially targeted. The grade coverage of each program was as 
follows: 



ESS: 


K-8 


IDP: 


4-8 


MINNEMAST: 


K-3 


SCIS: 


1-6 


S-APA: 


K-6 



The concepts and processes covered by the programs marked a significant 
departure from the traditional content of elementary science programs. The 
developers of all of these new programs believed that meaningful science 
learning would not occur If the focus was on having students memorize the 
products or output of science— the facts, laws, hypotheses— that are produced 
by scientists. Instead, they argued that, children should learn science It- 
self— the skills of observing, measuring, Inferring, predicting and testing 
employed, as well as the major concepts used by scientists to explain and 
organize facts and theories. These concepts were more than new facts to be 
memorized; they were mental "pegs" by which the students understand nature. 
Concepts or conceptual schemes Included were: organisms, ecosystems, systems, 
variables, conservation of energy, the statistical view of nature and the 
particulate nature of matter. 

Gathering the Information . Following the selection of the programs, the 
collecting of detailed Information about each of them began. As It turned 
out, this step proved to be complicated. All of the major curriculum deve- 
lopment projects (In all subject fields) are not well set up to handle dis- 
semination. After a short time they usually are so bombarded with requests 
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for Information that a public relations officer is required. Except for 
writing a few brochures and newsletters, handling site visitors, and attending 
conventions, these men are usually more concerned with keeping detailed 
information about the projects from getting out than vice versa. There seems 
to be almost a veil of secrecy which drops over the projects. The reasons 
for this secrecy are complex--probably including the difficulty in keeping 
up with changes in a rapidly moving development project, the large amount 
of time required to answer detailed questions, and, perhaps, the desire to 
curtail information about certain "trade secrets" used by the development 
that others might lift and thereby weaken the project's uniqueness. In the 
particular case at hand, the developers were also concerned about the ability 
of the laboratory, at the time a relatively unknown organization, to describe 
their project adequately. 

In any event, this step was more difficult and time consuming than ori- 
ginally anticipated. Information from the projects frequently was hard 
to get, sometimes inconsistent, usually Incomplete, and In many cases diffi- 
cult to understand. The original judgment that school people would have 
difficulty in learning about the projects In sufficient detail to adopt or 
reject them seemed to be confirmed. 

Legal Constraints 

Copyright constraints . One area In which there were potential legal 
constraints concerned the copyright laws as they related to the use of 
materials already copyrighted. At this stage the program staff had to 
determine its obligations to respect the copyrights of the developers' 
materials. Considerable thought was given to this problem and It was 
resolved in the following manner: 
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Although the new revision of the copyright laws was still In Congress, 
It was believed the issues that were holding uff*the bill were not related 
to the matter at hand and so It was hoped that most relevant sections of 
this bill would remain intact. Specifically Section 107 of the proposed law 
seemed relevant: 

Notwithstanding the orovisions of Section 106, the fair use of a 
copyrighted work, including such use by reproduction and copies 
or phono records or by any other means specified by that section, 
for purposes such as criticism, comment, news reporting, teaching, 
scholarship, or research, Is not an Infringement of copyright. In 
determining whether the use made of the work In any particular case 
Is a fair use factors to be considered shall include— (1) the 
purpose and character of the use, (2) the nature of the copyrighted 
work, (3) the amount and substantiality of the portion used In 
relationship to the copyrighted work as a whole, and (4) the effect 
of the use upon the potential market for or value of the copyrighted 
work. 24 



Given these limitations on exclusive rights, it seemed that selection 
of portions of curricular and promotional materials from the programs being 
reviewed and reproducing them in the Information unit came within the scope 
of criticism, comment, reporting, teaching, schlorship and research and 
was therefore not an infringement of copyright. The "determining factors" 
seemed to be as follows: 

(1) The Purpose and character of our use was clearly non-profit--a 
purpose traditionally recognized by the copyright office and specifically 
acknowledged in the judiciary committee's report on the copyright law. 

(It should be noted, though, that in their discussion of "purpose and 
character," they also mentioned the spontaniety of the act, the fact of a 



24y. s. Congress, House Committee on the Judiciary, A Bill for the 
General Revision of the Copyright Law, Title 17 of the United States Code , 
and for Other Purposes (H. R. 4347, 89th Congress, 2nd Session, October 12, 
1966). pp. 80-81 . 
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single copy rather than multiple copy, the fact of copying from a collection 
rather than from a single issue as being legitimate exemptions— none of which 
were applicable to our situation.) 

(2) Nature of the copyrighted work --Because we were not copying material 
that was consumable (workbooks, exercises, etc.) we probably qualified for 
exemption under this factor. (Interpretation of this section was not easy 
because it overlaps with the following one.) 

(3) Amount used- -A principle basis for our fair use claim was the 
fact that we had not copied the entire work or anything resembling the 
entire work. We merely extracted certain sections and elements that seemed 
appropriate to our work of criticism and reporting (the old cliche about a 
paragraph being the maximum amount that one can copy is not an explicit 
criterion of the proposed law). 

(4) Effect of use— Finally, in copying we had clearly not affected the 
potential market or value of the work by removing such substantial portions 
that our copies could replace the original work. The products of IDP, SCIS, 
etc. are teaching-learning products and one could hardly argue that the 
information unit replaces the original in such a way as to enable someone 

to teach from it. In fact, under most circumstances our copying should 
increase, not decrease, the copyright holder's sales. 

As a result of this analysis, therefore, it was concluded that we did 
not have to be concerned with legal constraints on our procedure of 
abstraction of small samples of copyrighted works for our comment and 
criticism of them for educational purposes. 
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Developing the preliminary form . Work on the preliminary form began 
late In 1967 and continued through 1968. The form developed consisted of the 
following: 

Level I . A slide/tape presentation of the overview. (A script for the 
overview film was turned over to a commercial film production company for 
completion, but It was not delivered in time to be used during the prelimin- 
ary field testing.) 

Level II . Various audiovisual forms were used to develop briefings on 
each of the programs included. The S— APA briefing was a black and white 
film produced by an outside film production company. The SCIS and IUP brief- 
ings were color slide/tape presentations with cartoon figures. The ESS 
briefing was available only in the form of a script. No "level two" presen- 
tation was available for the Minnesota Mathematics and Science Teaching Pro- 
ject. 

Level III . Printed drafts of reports on all programs were developed. 

The organization of these drafts was as follows: 

History 

Rationale 

Objectives and Goals 
Curriculim Organization 
Educational Processes 
Requirements for Implementation 
Effectiveness and Evaluation 
Appendix 

Level IV . Background Information was Knitted In the preliminary form. 

The purpose in the variety of media formats to the preliminary field 
test was to provide an opportunity to examine which media would be most 
acceptable to users. 

Where possible all of the materials were reviewed by the project's 
personnel for accuracy. 




Pictures of the mockup and preliminary forms are shown in figures 3 and 4. 




Figure 3 

The Mockup of the Elementary Science Information Unit 




Figure 4 
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The roockup was developed to give the preliminary test subjects a view 
of the overall design of the product as It was projected for final form. It 
was believed that this form was needed in view of the diverse media used In 
the preliminary form and the dlfficui'/ the subjects likely would have In 
envisioning the completed product. 

The preliminary field test . The preliminary field test took place on 
May 16, 1968, with 19 field test subjects from the Bay Area. They were se- 
lected to represent a cross-section of teachers, central staff curriculum 
coordinators and administrators as well as a cross-section of public and 
private schools. The test was administered in the Laboratory. 

the general questions that the preliminary field test was designed to 
answer were these: 

Is the project feasible? 

Are the needs for the product real? 

Is the proposed model likely to be successful? 

To get answers to these questions the following tests and prodedures 
were used according to the time schedule indicated. 

1 week before Subjects completed a form by mail identifying their 

arriving educational background, present responsibilities, 

current role in curriculum decision-making, attitude 
toward curriculum reform, and current knowledge of 
the science programs to be included in the informa- 
tion unit. 

9:00 a.m. Subjects were given an oral explanation of the work 

of the Laboratory. 

A two-page pretest was administered; the questions 
focused on the attitudes toward reform of the ele- 
mentary science curricula, current evaluation of the 
difficulty m getting information about science pro- 
grams, and the subjects* current knowledge of the 
new programs to be described in the information 
unit. 
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9:30 a.m. Subjects were shown the audiovisual overview. Imme- 

diately following, they completed a four-page ques- 
tionnaire designed to assess their attitude toward 
the presentation, changes in their knowledge and 
their retention of certain points of the presenta- 
tion. 



10:15 a.m. Break. 

10:30 a.m. {With a one-hour lunch break.) 

Subjects were randomly divided into two groups— 
half saw the audiovisual and read the reports on 
two programs (S--APA and MINNEMAST) and the other 
half saw the audiovisual (script in the case of 
ESS) and reports on the other two programs (ESS and 
SCIS). Immediately following each presentation and 
after reading each report, subjects responded to 
brief (three page) questionnaires directed to gen- 
eral reactions, attitudes, and knowledge. 

2:30 p.m. The entire group reconvened to take a posttest 

questionnaire and to engage in a group discussion. 

Conclusions . As indicated by the schedule, a great deal of data was 
collected during the field test. A detailed presentation of the results will 



not be made here. But the following conclusions were reached as a result of 



the analysis: 

First, it appeared there was a real need for the product. Pretest 
knowledge about the programs was very low— only a few of the subjects even 
knew of the existence of most of the programs— none knew of all of them. 

They all agreed that information about the programs was difficult to get and 
was needed. 



Second, the proposed format seemed successful. A number of changes were 
recommended (for example, the inclusion of certain information that had been 
omitted and re-emphasis on other types of information), but overall the 
product seemed to be highly valued and liked by the target group. 
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Third, because only a slight difference could be detected between 
reactions to the film and to the slfde/tape formats and the staff's 
knowledge of the differential in cost and time In producing films, a 
decision was made that during the main form development an effort would 
be made to develop the audiovisual component of the information unit In 
a slide/tape format. 

These conclusions, of course, were relatively subjective, given only 19 
subjects. But the staff felt confident that a satisfactory exposure of the 
proposed product had been made to a representative target group and that 
main form development should begin. 

In addition to the testing described above, the preliminary field test 
subjects were used for some more detailed research Into the question of the 
type and nature of information that various school personnel need for curric- 
ulum decision-making. This research Is a good example of the way in which 
applied research can be balanced with development In a product-oriented pro- 
gram. As noted elsewhere, a sufficient research base did not exist at the 
time this development was started. Rather than build one as a foundation 
before beginning development, however, the strategy selected was to couple 
research on critical questions with actual development. (For a more detailed 
explanation of this strategy see the article by Borg. 25) The questions 
asked in this research problem were: What Is the relationship between 
the type of job function and the type of Information needed? Whav is the 
relationship between where a decision-maker is in the decision phase and the 



* 8alance Between Educational Research and Develop* 

ment: A Question of Strategy," Educational Technology . VI, No. 7 (1969, 
pp. 5-11. ” ~ 
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type of Information needed? And, what is the interrelationship, if any, 
between job function and decision phase in terms of information needed? 

Stated more simply, we guessed that if a decision-maker was a superintendent 
of schools he had different information needs than a teacher; we also guessed 
that if a decision-maker was just beginning the process of looking for new 
ideas he had different information needs than someone close to the final de- 
cision point; we also wondered if there was any interaction between these 
two variables. 

To test our ideas, we used the same subjects involved in the preliminary 
field test. At the 9:00 a.m. meeting on May 16 we asked them to take a pretest. 
The test included a number of questions a decision-maker might ask about educa- 
tional development (e.g., "Does this program have explicit, clearly stated 
goals?" or "Is openness to new ideas or other personality styles essential for 
teachers?"). The instrument was pretested before use in the field test. 

The questions were grouped into nine classes or types of information. The 
subjects were requested to respond to each item in terms of how important 
the information was to a specific time in his decision-making. The subjects 
were divided randomly into two groups by job classification. One group was 
instructed to evaluate their information needs as though they were “search- 
ing for something better"; the remaining half completed the form with the 
instruction that they had narrov/ed their choice to one alternative and were 
"planning for trial adoption." At the 2:30 p.m. session the same ques- 
tionnaire was administered and the directions were switched for the two 
groups. The design for data analysis involved the use of Lindquist's Type 
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IV design.^ 6 Results indicated no relationship between decision "set" and job 
classification. There were significant interactions between decision-making 
set and subjects and between item class by person. The decision set by item 
class was not significant. The conclusion was that "there is no evidence 
of a difference in the overall value assigned in the two decision sets [but 
that] there is clearly a difference in the value or importance assigned to 
different classes of information."^ 7 In short, there were large individual 
differences which obscure any difference in job or exposure to curriculum 
information. Item analysis suggested that most information items are more 
important when the set is "planning for adoption" than when "searching for 
something better." The item analysis also revealed that the nine informa- 
tion classes could be rank ordered, and, although the differences among the 
classes were not large, they were consistently larger than variances within 
classes. The results of this small research project led the staff to con- 
clude that It would be fruitless to concentrate on differences in terms of 
types of Information required at different points along a decision continuum 
until better measures of controlling or predicting individual differences 
could be achieved. It was also concluded that organizing the information 
unit for groups of people at different job classifications was not warranted. 



26e. F. Lindquist, Design and Analysis of Experiments in Psychology and 
Education (Boston: Houghton Mifflin Co., 1953), pp. 285 ff. 

^ 7 Paul D. Hood, "The Perceived Need for Information About Education 
Developments" (unpublished paper, Far West Laboratory for Educational Re- 
search and Development, 1969). 




It was also decided that In the main field test focus would be placed on 
trying to identify variables from which individual differences might be pre- 
dicted. 

Personnel . The staff that had been used for the planning and conceptual 
stages continued with the project during the preliminary stage. A project 
director was hired. Holding a Ph.D. , he had experience In communications 
research, mass media, and Instructional technology. An additional person 
with an M.A. was added with a teaching (foreign language) background. 

Time and Costs . The preliminary form of the Information unit was 
developed and field tested between December of 1967 and May of 1968. The 
total costs for this stage were approximately $45,000. A more detailed 
breakdown of this cost follows: 



Consultant Review 


$ 1 ,300 


Information Collecting 


4,000 


Planning 


1,000 


Developing and Form 


25,500 


Testing and Data Analysis 


4,500 


Reporting (on the testing) 


700 


Overhead 28 


8,000 


Total 


$45,000 



The personnel costs represented 49% of this figure. 29 Film and other media 
supply costs were $11,500, including an $8,500 subcontract for one film. 



pO 

co 0verhead Included all Laboratory Management support but does not include 
such indirect costs as rent, utilities, communications, etc. These Indirect 
costs are prorated across the various tasks. 

^Personnel costs include salaries, wages and benefits of all permanent, 
part-time, and consultant staff. 
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STAGES OF DEVELOPMENT 



Stage 3: The Development and Field Testing of the Main Form 

Purposes of this stage . After the conclusion of the preliminary field 
test, the product was completely developed in its "main form." After develop- 
ment, it was field tested. The Laboratory's description of these two phases 
of Stage 3 are as follows: 

[In the first phase], preliminary product revision , any neces- 
sary changes to insure the effective use of the product In actual 
school use are made. Decisions about the changes to be made are 
based on the evaluation judgments and suggestions made by non- 
Laboratory participants and the experiences and observations of 
the staff in the preliminary testing stage. Occasionally, these 
revisions may be so extensive that they amount to a virtual re- 
design of the product so that a second preliminary field test is 
necessary. 

[In the second phase], ma in field testing , the product is 
tested, using larger samples of representatives of the intended 
audience in actual working situations. The evaluation is con- 
ducted quite systematically and is designed primarily to provide 
information on the product's effectiveness in achieving the stated 
objectives. The main field test is also used to identify ways In 
which parts of the product might be improved. Generally the staff 
responsible for the development of the product is actively Involved 
in the field test as observers of the process and as coordinators 
of the field-testing activities of the participants. An additional 
purpose of this main field test is to identify points at which the 
users of the products need more, or more specific, directions for 
product use to insure its effectiveness. Following the main field 
test, decisions are made about possible modifications of the pro- 
duct necessary to correct any deficiencies identified during the 
evaluation. If the revisions in the product which are made [at 
this phase] are quite extensive, the main field test may be subse- 
quently repeated to determine the effectiveness of the revised 

product. 30 

The case study of the Elementary Science Information Unit at the "main 
form" stage continues under the following headings: Contacts with Developers, 



Of) 

Far West Laboratory for Educational Research and Development, 
Contractor's Request. 
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Development of a Test Plan, Development of the Components, Testing, Evaluation, 
Conclusions, Personnel, and Costs. 

Contacts with developers . Drafts of all the materials developed at the 
preliminary stage were sent to the respective science project directors for 
comments. New information about the programs continued to come in. It was 
clear that one of the major problems was that the science programs had all 
undergone major changes requiring redrafting the reports. 

By this time, the developers were much more cooperative. Though they 
may not have fully understood or agreed with the objectives of the project, 
they could now see that the information released would be of a significant 
nature and so they were willing to provide help. 

Development of a test plan . Since detailed testing of the product 
is the key feature of this stage of development, a major plan for field test- 
ing was devised. This plan included a detailed specification of objectives, 
the design of instruments that would be used, and a test administration plan. 

The objectives were classed into three groups (decision objectives, 
information objectives, affective objectives). General goals were translated 
into statements of user performance; conditions were specified and standards 
set. An abridged statement of these objectives follows this paragraph. It 
should be noted that one objective that might have been used--the number of 
adoptions made as a result of the use of the information unit--was not used. 

The rationale for this exclusion was s that a well-informed user might choose 
not to adopt any of the programs; all of them might fail to meet his objec- 
tives and constraints. To require the information unit to effect adoptions 
would be to confound the quality and nature of the innovations themselves 
with the information unit. As indicated earlier, the goal of the informa- 
tion unit was to induce rational decision-mak1ng~-not adoption of new science 
programs . 
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1. DECISION OBJECTIVES: {Does the Information Unit enable 

school personnel to make decisions they find acceptable?) 

a. Having used the Information Unit, subjects will rate 
each program on a seven-point scale according to the degree 
that it fits their goals and resources. Over 80$ of all 
subjects will be able to do this under " performance " con- 
ditions. 

b. Having used the Information Unit., subjects will Indi- 
cate which of the programs their schools would consider 
for adoption. Over 50$ of the subjects will make definite 
decisions adopting or rejecting all programs under "opera- 
tional" conditions. 

c. Having used the Information Unit, subjects will not 

Indicate: (1) they did not need the Information Unit, or 

(2) they would be unable to make a decision because their 
goals were unclear or because they would need to check on 
the accuracy of the Information In the Information Unit. 
Instead, they will Indicate that the Information Unit helped 
them either make a decision or narrow their choice to one 

or two programs. Over 60$ of the subjects will check the 
positive "decision" categories from a multiple checl: list 
provided under "operational" conditions. 

2. INFORMATION OBJECTIVES: (Does the Information Unit Impart 

the basic Information needed to make decisions?) 

a. After using the Infonnatlon Unit, subjects will not 
Indicate the need for any additional information on any of 
the programs. 

b. After using the Information Unit, subjects will show 
a statistically significant (.05) Increase In their own 
estimates of their knowledge about the various programs. 

c. After using the Information Unit under "performance" 
conditions, subjects will correctly match statements about 
the programs with the names of programs. Eighty percent 
of l,Se subjects will correctly match Items they consider 
Important In making an evaluation of the programs. 

3. AFFECTIVE OBJECTIVES: (Do the users like the Information 

Unit, find It useful, prefer It to other sources of Information, 
and would they use It again or recommend It to others?) 

a. Given an attitude questionnaire (semantic differential 
format) after using the Information Unit, subjects will 
express a positive attitude toward the Information Unit 

on scales such as "useful , Interesting, satisfactory, 
complete, reliable, easy to use, well -organized, and 
clear." The mean average for all subjects will be 4.0 
or better (7.0 being the positive end of the scale). 

b. Given a list of possible primary and secondary sources of 
information about new science programs, after using the Infor- 
mation Unit, subjects will Indicate the Information Unit Is 
superior to all secondary sources. Sixty percent of the sub- 
jects would rate the Information Unit above all secondary 
sources. 31 
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(A description of the procedures used to select subjects and details of 
the test administration are found below under "testing.") 

D evelopment of the components . Between June 1968 and December 1968 the 
main form of the information unit was developed. This form consisted of the 
following elements: 

Level I: The Overview film which was commissioned for the preliminary 
field test but which was not delivered on time became available for the main 
form. During the actual field testing, however, it became clear that the 
film v/as quite unsuccessful: viewers seemed antagonized by some of the mate- 

rial and affective evaluation data indicated that the film was falling far 
below the standards set for It. In retrospect, the problem was caused by the 
low quality film production and an apparent oversimplification of the prob- 
lems of science education. The staff believed that continued use of the 
film In the testing program might lower the results of the overall product 
and damage the Laboratory's reputation with the schools Involved. The film 
was, therefore, dropped from the main form. Fortunately there was enough 
time to develop an alternative print form, a booklet divided into two 
sections: "A User Guide" and "General Summary." Ihe first section re- 

placed the film’s Instructions on how to use the Information unit and the 
sutmary presented a discussion of the trends of the new programs In a 
form modified from the film presentation. 

The changes which occured In the product at Level I during this field 
test provide a good example of the problems confronting a developmental 
effort that require strategies different from those of research programs. 

In a research design the changing of the "treatment" halfway through the 
test would be unacceptable. However, adherence to strict research 




37 



methodology could not be followed In a development project because of 
the constraint of using voluntary field test subjects in "real" situa- 
tions and because of the costs. Fortunately with a phased test administra- 
tion (not all tests were conducted at the same time) and a continuous moni- 
toring of the results at each site, improvements in the product could be 
made as the testing program went along. The problem is similar to the one 
faced by any curriculum development: because the testing situation is 

"real" and students are dependent upon the program for Instruction, changes 
must be made as better methods become known or deficiencies identified. 

Level II: The media forms of the intermediate level of the Information 
unit continued to be varied during the main form. The briefing on Science-- 
A Process Approach was still a black and white film. The slide presentation 
of the Inquiry Development Program used In the preliminary form was changed 
to a film. It is worth noting that this film form was somewhat experimental} 
It might have been better called a "fllmograph" technique. Although the 
release form was 16mm color film, the photography Involved a combination of 
original 16mm color "action" footage and prints of 16nm color slides shot on 
the same location. During the field tests few subjects could detect that this 
was not the usual, complete 16mm film; only media experts noticed the 
difference. The result was a considerable saving In dollars. A complete 
20-minute, color film shot on color film stock would have cost ten to 
twenty thousand dollars. (A thousand dollars a running minute is a rough 
Index of the cost for a typical educational film.) The 1DP film cost about 
$5,000. The Science Curriculum Improvement Study slide/tape was converted 
from cartoons to "real" pictures of classroom settings. This change 
resulted from a number of torments about the unsuitability of the cartoons 
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In the preliminary field version. Although these comments did not come 
from a majority of the test subjects, the staff felt it would be better 
to go to a form that was acceptable to all of the subjects. The Elementary 
S cience Study was also presented in slide/tape format. No audiovisual 
presentation on the Minnesota Mathematics and Science Teaching Project was 
available. An attempt had been made to shoot some original stock which could 
be used fn a fllmographic approach for MINNEMAST, but because of the loca- 
tion of the shooting (Minnesota), the laboratory staff did not have tight 
control over its quality. The problem was further complicated by the un- 
certainty about the continued funding of the MINNEMAST project. It was 
therefore decided to Include only the written report on MINNEMAST at this 
stage. 

Level 111 and IV: Levels III and IV were combined (background material 
was Inserted as an appendix into the reports). Reports were revised to make 
them less wordy. A slightly modified organization was adopted to conform to 
suggestions picked up during the preliminary field tests. The chapter organi- 
zation of the main form of the reports was as follows: 

Organizational Background 

Theoretical Background 

Content/Materials/Organization 

Teaching/Learning Strategy 

Implementation 

Evaluation 

A photograph of the elements of the Elementary Science Information Unit 
in Its main form appears In figure S. 





Figure 5 

The Main Form of the Elementary Science Information Unit 
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Testing . A relatively rigorous site selection procedure was used. A 
randomly selected sample of one-fourth of all the elementary school super- 
intendents In the public and private schools In Northern California and 
Nevada were Invited to participate In the field test. Schools responding 
were divided Into four groups based upon two types of conditions for two 
variables that were thought to Influence the performance of the Information 
unit. One variable was wither or not the schools were obliged by state law 
to use a specific text for elementary science (California has such a law, 
Nevada does not) and the other variable was the degree to which the geographic 
and demographic characteristics of the site suggested It was ’’remote" as 
opposed to being an area where cormiunl cation was "maximum." Remoteness was 
defined In terms of distance from an urban center and from c college 
or university. Twice as many urban as rural sites were selected to corre- 
spond to the true ratio of these characteristics In the area covered. 

It was estimated In advance that an average of seven subjects would be 
Involved at each cite. The table below describes the relation of subjects 
to sites as they were to be distributed. 



MAIN FIELD TEST SITE SELECTION PLAN 



URBAN RURAL 

"Co-nmunl cation Maximum" "Conminl cation Minimum" 

State Text State Text State Text State Text 

R equl red Not Required Required Not Required 

No. of 

Sites 6 6 3 3 

No. of 
Respon- 
dents 42 42 .>1 21 
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During actual field testing, 19 sites and a total of 181 subjects became 
involved. The additional school and subjects were added because one school 
which we originally thought would participate, backed out; a substitute was 
selected and the original school then asked to be included. The exact number 
of subjects also varied because we could not control the number of subjects 
each superintendent selected at each site. The subjects Included 134 elemen- 
tary teachers, 28 administrators, 9 curriculum consultants, 3 lay advisory 
people, 2 secondary teachers, and 5 Individuals who classified themselves 
Into more than one of those categories. One hundred twenty-eight subjects 
were from California Public Schools, 10 were from California Private Schools, 
and 42 were from Nevada Public Schools (one respondent was unidentified). 
Eleven subjects were from isolated (rural) schools; 169 were from non- 
isolated (urban) settings. 

Evaluation . In order ^o allow the reader to compare the results of the 
main test with the results of the operational field test, presentation of 
results has been omitted here and ti. data Is reported In a coordinated 
fashion In the "evaluation" section of the chapter entitled: "Stage 4: 

The Development and Field Testing of the Operational Form." (See oage 49) 
Conclusions . The general conclusion reached by the staff as a result 
of the main field test was that the Information unit had met all of Its 
objectives satisfactorily except In the area of knowledge retention. With 
hindsight the staff recognized that part of this failure was due to Inadequate 
control over the amount of time spent reading the detailed reports. Since 
this control could not be enforced during the operational testing or In 
final release, It was decided to place more emphasis fn the Level I material 
on those Items of Information that were thought to be most critical. Several 
field test subjects suggested that this might be accomplished with more 
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chart-like analyses. It was also clear from the staff observation of the 
field tests that the printed Level I material did not work well In group 
settings. It was thought that audiovisual materials should be re-developed 
at this level. (The original decision to drop audiovisual at this level was 
based on the poor performance of the film that had been developed, not any 
Inherent difficulties In the medium.) 

Analysis of the results of the Level II audiovisual briefings revealed 
no differences In attitude, decision-making function, or Information capacity 
of the filmed briefings as compared to the sllde/tape ones. Given the differ- 
ence In cost becween the two. It was therefore decided to go with the s'lde/tape 
format, later converted to filmstrips. This was probably the most Important 
physical change In the model that occurred anywhere along the developmental 
cycle. 

During the time that passed between the drafting of the main form reports 
and the scheduled date of the operational field test. It also became known that 
re*.. Ions In the reports would be necessary to make them accurate. A detailed 
analysis of the value placed on sections of the reports also revealed that 
the "historical background" section was least Important and so it was placed 
at the end of the report rather than at the beginning. 

Personnel . The personnel Involved In this stage remained the same as 
at the preliminary stage, with these exceptions: one clerk typist was 
added to the staff to handle the additional level of paper flow In making 
revisions In reports, and the audiovisual-production staff of the Laboratory 
was used heavily to produce the audiovisual components of the package after 
project staff drafted scripts. 
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Time and Costs . The development and testing of the main form took 
place between June, 1968 and February of 1969. The total costs for this 
activity were $46,200. A breakdown of the costs Is as follows: 



Revising and Developing the Form $19,000 
Field Testing and Data Analysis 19,000 
Report Preparation 700 
Overhead 7,500 



Total $46,200 



Personnel costs represented 49X of this amount. Media production costs were 
approximately $6,000. 
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STAGES OF DEVELOPMENT 



Stage 4: The Development and Held Testing of the Operational Form 

Purpose of this stage . As defined by the Laboratory, the next phases 
of development include: 

Main product revision , which usually Involves minor modi- 
fication of the product and the development of auxiliary ma- 
terials necessary to Insure that the product will be entirely 
self-sufficient when put In operational use. As noted above, 
occasionally the product revisions are so extensive that a 
second main product test ... has to be conducted. 

[And,] operational product testing , the materials and 
processes which constitute the product are tested In actual 
use In classroom or school situations without the participation 
of the staff responsible for the product's development*, that 
Is, the product Is tested in the completely realistic setting 
for which It Is ultimately Intended. The primary purpose cf 
the operational test Is to determine If the product can be used 
widely In schools without the active Intervention or partici- 
pation of th«? staff. This phase Is crucial In the development 
and definition of a product of the Laboratory. The Laboratory 
cannot and does not wish to provide a service function In con- 
nection with Its products. Rather, the goal of the laboratory 
Is to produce completed products which have been thoroughly 
tested for use by school personnel without any active partici- 
pation by the Laboratory staff. 32 

The following section will discuss these phases under the headings: 
contacts with the developer, development of a te^t plan, development of com- 
ponents, testing, evaluation, conclusions, personnel, time and costs. 

Contacts with the developers . As the project continued, the cooperation 
of the developers grew. It was at this phase that one of the developers 
called, after receiving a revised draft of a report and viewing a filmstrip, 
to say that the Laboratory staff had done a better job explaining his project 
than his own staff. Similar comments were made by other developers and to 
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date three of the five projects have requested permission to use the Labora- 
tory's copyrighted material In their dissemination effort. (Permission has 
been refused, to maintain the Integrity of the Information unit as a decision- 
oriented package describing all the programs.) 

Development of a test plan . The objectives of the Information unit 
remained unchanged throughout this stage. Particular emphasis was placed on the 
terminal objectives (decision-making), the affective objectives, and data 
about the use of the unit under operational conditions. Instruments similar 
to those used In the main field test were retained. No detailed sampling 
plan .:as specified. 

Development of Component s. . 

Level I: The "User Guide and General Summary" was revised according 
to the decisions made at the conclusion of the main field test. They were 
renamed "The Screening Guide" and the "Guide to the Selection of an Elemen- 
tary Science Curriculum." These were printed as separate booklets. In addi- 
tion, an audiovisual filmstrip/tape "Preface" was developed for the operational 
form according to the decisions discussed In the "conclusion" section of the 
iwlr. stage of development. 

Level II: All of the audio briefings were converted to filmstrip/tape. 

Careful examination of the distribution problems convinced the staff that 
circulating slide sets was very cumbersome and likely to produce problems In 
operational use. 

Level lit: (Level !V--was dropped; essential elements were spread 

throughout other levels of the product.) The reports were revised again and 
the chapter headings reordered (historical background section last* goals 
end objectives first) and their names altered. 

A box design for the unit was developed and executed. 

A photograph of the operational version appears In figure 6. 




Figure 6 

The Operational Form of the Elementary Science Information Unit 
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Testing . The sample for this field test was not drawn as rigorously as 
the main field test sample since no significant basis could be found for 
distinctions between sites of different characteristics. Sites expressing 
Interest In the project were contacted by letter and asked to use and 
evaluate the operational field test ( n FT) version. (Most of these expressions 
of interest were obtained while selecting main field test sites.) Sites were 
selected to represent different geographic characteristics and a variety o’’ 
uses. Approximately one- third of the sites was drawn from the St, Louis 
area through the cooperation of the Central Midwestern Regional Educational 
Laboratory in St. Ann. All other sites were in the geographic area covered 
by the Laboratory. 

As much as possible, only sites designating themselves as actively 
involved in the review and evaluation of innovative alternatives in science 
education were selected. No restrictions on the selection of participants 
were set, but all sites submitted data indicating the positions of those 
involved and the review procedure(s) followed. 

At each site, one participant completed the questionnaire for the 
group involved in reviewing the package. Thus, for 24 groups, a total of 
24 respondents (teachers, principals, central office staff, teachers in 
training, professors, and researchers) completed the field test instrument. 

The coordinator at each site was required to answer a three page 
questionnaire when the use of the information unit was complete. In addition 
the coordinator at all sites within the Laboratory’s region and those schools 
cooperating from the St. Louis area met with a Laboratory staff member for 
an hour interview and responded to questions contained in an interview 
guideline. The questionnaire included questions asked during the main field 
test to test decision, information, and attitude objectives for the information 
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unit. In addition, respondents were asked to describe how they used the 
product and to indicate various reactions to it. (Complete copies of the 
Instruments can be had on request.) 

No direct controls over the use of the materials were exercised. All 
groups were mailed one copy of the information package for use during the 
months of May and June, 1969, and asked to use it in a v.av suitable to their 
needs. 

The only restriction placed on the use of the box was that each site 
was limited to a one-month use and agreed to provide feedback data after 
using the package. The first "test" of the package was whether sites would 
return it unused at the end of the time period. This did not happen; all 
participating groups made use of the information unit. By itself, the use 
of the information unit by all users is a significant piece of evaluation 
information. 

Evaluation . Results of the operational field test are reported below 
as abstracted from the final report of the product development cycle. 33 For 
the reader's convenience, the comparable data from the main field test 
{"performance field test"— PFT) has al r o been included. 



33 
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1. DECISION OBJECTIVES: Does the IU (information unit) enable 

school personnel to make decisions they find acceptable? The field 
tests indicate that it does. The performances that were required 
during the various tests and the results of this testing are as 
follows: 



a. Having used the 10, subjects will rate each IU program 
reviewed (SCJS, ESS, etc.) on a seven-point scale according to 
the degree that fits their goals and resources. They will not 
indicate they have insufficient information. During the Main 
Field Test (MFT) 87% of all subjects were able to do this on 
all programs. [See Table 2.] 

b. Having used the IU, subjects will indicate which of 
the programs their schools would consider for adoption. During 
the Operational Field Test (OFT) over 57% of the subjects 

(the percentage ranged as high as 87% for one particular 
program) made "adoption-rejection" decisions on all programs. 

[See Table 2.] 

c. Having used the IU, subjects will not indicate that 
they did not need the IU, or would be unable to make a decision 
because their goals were unclear or because they would need 

to check on the accuracy of the information in the IU. 

Instead, they will indicate that the IU helped them either 
make a decision or narrow their choice to one or two pro- 
grams. During the MFT, 58% of the subjects checked the 
positive "decision" categories from a multiple check list. 

During the OFT. 63% of the subjects fol lowed suit. [See 
Table 3.] 

In short, it appears that the majority of all the test res- 
pondents could make satisfactory decisions with the IU. 

2. INFORMATION OBJECTIVES: Does the IU impart the basic information 

needed to make such decisions? The answers to this question are con- 
flicting. On the one hand, subjects were asked if any information they 
needed to make decisions was not supplied. Almost no one indicated 
the need for additional information (less than 5%). When such a need 
was expressed, it was usually for information that was omitted from 
the IU because it was not available (but would be included later) or 
because the constraints of development had set it outside the boundaries 
of what was to be included~«such as information from "early adopters." 
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TABLE 2 



This table presents the percentages of the operational field 
test respondents who made definite commitments about their choice 
of programs and the percentage of the main field test participants 
who were asked to make judgments about the "fit" of each program to 
their own school's goals and resources. (Clearly, the former Is a 
more "difficult" Item.) 



MAIN FIELD TEST OPERATIONAL FIELD TEST 





(n = 124- 


■120) 


(n - 


23) 




% PFT 


% Checking 


% OFT 


% 




"Matching" 


Insufficient 


Judgments 


of Non- 


Program 




Information 

1 


Made 


Response 


SCIS 


93% 


7% 


57% 


43% 


S--APA 


87 


13 


78 


22 


I DP 


91 


9 


87 


13 


ESS 


94 


6 


65 


35 



Table 2 suggests that while most operational field test respondents 
could make a judgment about how their schools would feel about each 
program, they were more reluctant to make a definite or precise judg- 
ment than main field test participants to estimate the "fit" of the 
program. 
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TABLE 3 



Main 

Field Test 
(n = 138) 



(a) 


Because of n\y previews knowledge 
and experience, I do not believe 
the IU would be of significant 
use In selecting a program. 


n 


(b) 


I already had a pretty firm Idea 
about the program T would pick 
and would not have needed the IU. 


i 


(c) 


I am not ready to make a decision 
at this time; our objectives must 


24 




be more clearly defined. * 




(d) 


Before I would make such a 
decision, I would want to do 
my own checking on the infor- 


12 




mation presented in the IU. 





(e) *The IU enabled me to decide that 0 

none of the programs (including 
the one I am now using) is satis- 
factory and I will continue to 
look for another one. 

(f) *The IU helped me narrow my choice 45 

down to one or two programs on 
which I would want more information. 

(g) *The IU was of definite help; on 13 

the basis of the information sup- 
plied, I would be able to pick a 
program to fit my needs. 

(h) The IU was of definite help in 4 

some other way. (Please specify: 



* * positive "decision categories", total: 0835 




Operational 
Field Test 
(n = 24) 

m 

8 

4 

4 

0 

42 

21 

8 

6335 
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It is noteworthy that operational field test respondents are 
more "ready" than main field test participants to use the IU for 
decision-making [re: alternative (c)l and have high confidence in 

the information provided [re: alternative (d)]. 

In contrast, however, they are initially more informed and 
therefore less in need of the IU than performance field test 
participants [re: alternatives (a) and (b)]. In part, the staff 
attributes some of this increase to the use of a single respondent 
summarizing the conclusions of the larger group of subjects rather 
than the completion of each questionnaire by all subjects. The 
respondent selected by the school usually was the best informed 
in the school and because of the use of the personal pronoun "I" 
in the questionnaire, he had a tendency to answer for himself on 
this item rather than considering the whole group. Though the 
same response set would be equally applicable to other items, it 
is this item where he was most likely to differ from the group. 

Further, in a supplementary study by Hood and Hutchins (1969), 34 
it was shown rather conclusively that the IU did result in statis- 
tically significant increases in the users' own estimates of their 
knowledge about the various programs. This study was a supple- 
ment to the main field test. Pre-post conditions prevailed, but 
subjects were volunteers and no method of random assignment of 
subjects to "treatment" and "control" conditions was possible. 

In contrast, however, when an instrument requiring subjects to 
match statements about each program with the naiues of the program 
(a posttest only for the MFT) only about a third of the subjects 
could pass more than half of the items. [See Table 4.] Since 
subjects were able to skip items that called for information that 
they thought unimportant, it seems relatively clear that not as 
much information could be recalled by the subjects as either the 
staff or the subjects thought should be recalled. Why? One 
reason is probably that subjects did not spend as much time with 
the Reports as they had been asked to. The other and more obvious 
conclusion is simply that the IU was not working as well as had 
been hoped. 

One of the reasons that users did not spend enough time with 
each booklet was that they are unwilling to reduce the number of 
programs they continued to consider after each level of the IU; 
therefore they spent less of their limited time with each program. 
This is an important variation from the intended three-level model 
of the IU. After viewing or reading each level, the user was 
supposed to eliminate programs so that by the time he arrived at 
level three (the Reports) he would have narrowed his choice to 
one or two programs that could be studied in depth. In fact, 
users were reluctant to follow this pattern. Apparently they 
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TABLE 4 

Percents and Cumulative Percents of Respondents 
Receiving scores from 04 to 100% on knowledge retention Items. 
Score * # correct * (# correct + # Incorrect) 

Scores Percent Cumulative % 

(% Correct) Respondents Respondents 



100% 


0% 


0% 


90-99% 


0% 


0% 


80-89% 


1% 


1% 


70-79% 


3% 


4% 


60-69% 


12% 


16% 


50-59% 


13% 


29% 


40-49% 


, 13% 


42% 


30-39% 


17% 


59% 


20-29% 


13% 


72% 


10-19% 


16% 


88% 


1-9% 


10% 


98% 


0% 


2% 


100% 
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felt obliged to look at all programs--so long as the total number 
to be considered remains within a manageable limit. (Experience 
with other Ill's and discussions with users has led the staff to 
conclude that when the total number of programs exceeds that 5-7 
range, the user Is prepared to narrow his choices earlier in the 
selection processes to a smaller number (2 or 3) than he will if 
he thinks himself capable of fully considering the total number 
Initially presented to him.) Much of this is speculation, however, 
and all that can be definitely concluded is that while the informa- 
tion objectives of the IU are not being met as successfully as 
Initially hoped, they are being met in part. 

3. ATTITUDE OBJECTIVES: Do the users like the IU, find it 

useful, prefer it to other sources of information, and would they 
use It again or recommend It to others? The answers to these 
questions speak positively for the value of the IU. 

a. (liven semantic differential instrument after using 
the IU, subjects shall express a positive attitude toward the 
IU on scales such as "useful, interesting, satisfactory, 
complete, reliable, easy to use, well organized, and clear." 

On the MFT all subjects averaged over 4.8, on all parts of 
the IU, on a seven-point scale (seven being the positive end). 

On the OFT subjects averaged over 5.9. [See Table 5.] 

b. Given a list of possible sources of information for getting 
adequate information about new science programs, subjects will 
Indicate that the III i*, among the most desirable sources. On 

both the MFT and the OFT the IU was ranked or rated above "hiring 
consultants, professional meetings or conventions, contacts with 
publishers and product developers, and reading professional jour- 
nals." It was ranked below "workshops using the new science cur- 
riculum materials, and site visits to innovative projects." On 
the MFT It was ranked below "conversations with professionals 
whose judgments I value"; on the OFT it ranked above the same 
item. In other words, the IU is ranked above all other secondary 
sources and rated below interpersonal contacts. [See Table 6.] 

c. As to whether subjects would use the IU again, only 
two of the 24 OFT respondents indicated they would not request 
the IU and one of those said he would request an IU in another 
field (since he presumably got what he wanted out of the present 
IU). All the OFT respondents said they would request an IU In 
other fields if they were available. 

Clearly, the IU has met with a favorable reaction from users. 

4. OTHER RELEVANT INFORMATION: In addition to the other evidence 

presented, the following pieces of information seem relevant to an 
evaluation of ti.e IU: 



TABLE 5 



Mean Scores Over Each of the Nine Scales 
for Both the Main and the Operational Field Test 



Scale 

Interesting 

Satisfactory 

Sufficient 

Complete 

Reliable 

Easy to Use 

Well -organized 

Clear 

Useful 



IU m IU OFT 

5.7 6.1 

5.2 5.9 

4.7 5.4 

4.3 5.2 

5.3 6.0 

4.9 6.4 

3.2 6.4 

4.7 6.0 

5.6 5.6 
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TABLE 6 



Usefulness Rating/Rank 



Source 


Main 

Field Test 
(r. = 131-141) 
Rating - Rank 


Operational 
Field Test 
(n = 24) 
Rating - Rank 


Workshops using new science 
curriculum materials 


6.50 


1 


6.50 


1 


Site visits to innovative projects 


6.2’ 


2 


5.75 


3 


Pilot projects in own district 


6.13 


3 


6.00 


2 


Conversations with professionals 
whose judgment I value 


5.97 


4 


5.58 


5-6 


Information Unit 


5.63 


5 


5.61 


4 


Hiring of consultants 


5.05 


6 


5.04 


8 


Professional conventions or meetings 


4.97 


7 


5.58 


5-6 


Contacts with publishers and 
product developers 


4.76 


8 


5.08 


7 


Reading of professional journals 


4.74 


9 


4.21 


9 



Another way to treat the operational data Is to look beyond "use- 
fulness" ratings and consider a ''consensus" score or the degree of 
agreement among respondents regarding sources they would prefer to the 
IU. On the basis of the number of respondents who place a source either 
"before" or "after" the IU in regard to its usefulness for curriculum 
decision-making, we can compute an algebraic score for a source which 
more precisely locates It In relation to the IU. Each operational 
field test respondent Indicated which of the sources was more valuable 
to him than the IU. Whenever a source was pointed out as more valuable, 
we gave it a score of +1; otherwise, it was given a score of -1. The 
sum of scores for each source (over the 24 respondents) is a positive 
or negative value, depending on whether more respondents like it better 
or not as well as the IU. The following exhibit presents the results 
of such analysis on the operational field test data: 
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+25 

-- Workshops using n^w science curriculum materials 

m* -- Pilot projects 
“5 - IU 

~ -- Site visits 



-- Conta* s with publishers and product developers 

— -- Conversations with professionals whose judgment I value, 
Tf5 Hiring of consultants 



^0 



Professional conventions or meetings 






This gives us quite a different picture of the IU in relation to 
these other sources. In terms of this "consensus" measure, only workshops 
are markedly more valuable than the iU fo; curriculum decisions. Pilot 
proje. and site visits are preferred about as often as the IU. De- 
velopers 1 report*, consultants, and Interpersonal conversation are not 
as often preferred, and meetings and journals are rarely preferred. 
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a. All OFT respondents indicated that they believe the 
Laboratory had given a fair review of the programs. 

b. When asked how much they would pay for renting the 
IU, 6 OFT subjects indicated they would pay $50 for a month's 
rental, 5 said they would pay $25 and 4 said they would pay $10. 
Twelve of the fifteen OFT respondents said they would pay $15 
for one week with the option of extending it to a month for 
$25. In order to buy the IU 6 respondents (of 13) said they 
would pay as much as $100; 7 said they would pay around $50. 
Overall, 10 respondents would prefer to rent, 9 would prefer 
purchasing a permanent copy. (The final selling price was 

$75; no rental plan was available.) 

c. OFT respondents preferred a noncommercial publisher or 
at least one not Involved with any of the programs. 

d. It seems probable that at least one-fourth of the 
secondary target audience for the IU (schools of education, 
state departments of education, and other linking agents to 
the schools) will be willing to purchase the IU. 

e. The primar> use of the IU outside the field testing 
situations has been as "an information source" or preservice 
tool rather than as 3 decision-making tool. (Out of 133 users 
of the IU since the completion of the OFT, 64 have used it 
"for information," 26 as a preservice tool, 19 as an inservice 
tool, and 14 as a decision-making tool.) 

f. University professors using the IU with their science 
education classes have been unanimous in their positive reac- 
tion to the IU. 

g. As of December, 1969, the Elementary Science IU has 
been tested or used in 33 states and 2 foreign countries. It 
has been seen or used by over 5,000 people at 91 sites. It has 
been used in institutions of higher education, in over 75 public 
and private elementary school districts, and in over 40 R A D 
laboratories, Title III Centers, and State Departments of 
Education. During all this time no negative feedback about the 
purpose or general nature of the IU has been received. 

h. In Juno of 1970 a telephone survey of a random sample 
of 1/3 of the schools participating in the main and operational 
field tests was made. The interview focused on gatheringg 
long range data about the use of the Information unit. The 
results indicated that 50? of the schools used the information 
to arrive at a "real" decision; the remainder were simply 
cooperating with the request to field test the unit and had not 
planned to review their science curriculum that year. Over- 
whelmingly the unit was rated good or excellent by all users. 
However' in only one situation (out of 20) was the unit al one 
accorded the primary role in leading to an adoption decision. 
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Elsewhere, the unit was used with additional Information from 
persons using or familiar with programs. Personal experience and 
assessment of the program appears to be crucially Important In 
making a decision of such magnitude. However, personal Interviews 
stressed the Importance of having Information gathered and 
organized to assist In the decision process as well as to Inform 
teachers In preservice and Inservice classes and workshops. 

Con clusions . As Indicated by the test data reported, the 
Information unit generally met all of its objectives. In summary, 
the evaluation revealed the following: 

- 87% of the subjects could rate each program reviewed In 
the unit against their goals and objectives to a high 
degree of confidence. 

- 57% (and In some cases many more) of the subjects could 
Indicate which of the programs they would adopt (or 
recommend for adoption), 

- 63% of the subjects Indicated that the Information unit 
provided all the Information they needed to make an 
adoption (or rejection) decision. 

- Subjects experienced a statistically significant Increase 
in their own estimation of their knowledge about the pro- 
grams. 

- Average ratings by subjects placed the information unit 
at 5.9 (toward the positive end) on a seven point scale 
combining such criteria as useful, Interesting, satis- 
factory, complete, reliable, easy to use, well -organized, 
and clear. 

- Given a list of possible resources of curriculum informa- 
tion, subjects indicated that the Information unit was far 
more desirable than a)J[ other secondary sources of comparable 
Information (hiring consultants, professional meetings or 
conventions, and Journals) and almost equivalent to all 
primary sources (workshops usi?ig the new science curriculum 
materials, site visits to innovative projects, and conver- 
sations with professionals whose judgments tney valued). 

The only negative information obtained during the field evaluations 
of the information unit was that only one-third of the subjects cculd 
correctly match the names of the various programs with facts that the 
staff Judged to be Important about each program. This low performance 
may be due to many users not spending as much time with the unit as was 
desired and having difficulty associating what they remembered about the 
programs with specific names of the programs (although they could, In 
general, remember the nature of the programs). Perhaps the more obvious 
conclusion is that users simply did not remember everything that the 
staff believed was Important to make a "good" decision. 
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As a result, the staff recoimended that the unit be released by the 
Laboratory. (Certain modifications were also recommended, see below.) The 
procedure used to secure this release requires that the Director of the Labor- 
atory select a panel to review the product and judge Its quality. These judges 
are drawn from the senior staff of the Laboratory— but outside the program In 
which the product was developed. This procedure was followed and the product 
was approved for release. 

The changes recommended In the released product form were these (a full 
rationale for these changes Is spelled out In the final product report): 

i 

1. It was recommended that one more program should be added to the 
five described by the Intormatlon unit: COPES— the Conceptually Oriented 
Program In Science. 

The Inclusion of this program after completion of the operational field 
test was dictated by the request of the American Association for the Advance- 
ment of Science that It be Included In our treatment of the "new elementary 
science programs." The existence of COPES was known to the staff throughout 
the developmental cycle of the unit but It had not been Included because It was 
not as far along In Its developmental cycle as the other programs. It seemed 
clear, however, that It would be completed shortly after the unit was to be 
released and therefore should be Included. It was tire opinion of the program 
staff responsible for the unit that the addition of this program would Improve 
the performance and operation of the product and would not be detrimental as 
long as the program materia) was accurate and exactly parallel In form to the 
iraterlal on the other programs. 

One other program, "Measure and Find Out," the outcome of a project 
entitled "The Study of a Quantitative Approach In Elementary School Science," 
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might have been Included In the final product but was not. The project went 
out of operation shortly before the developmental cycle of the Information unit 
began, and It was not known until after distribution of the unit had been 
arranged that this particular science program had been picked up by a commer- 
cial publisher for distribution. Omission of this program was not considered 
critical because It Is only a supplementary, one-year science program and does 
not have the multigrade characteristic of most of the other programs. {In 
all fairness, It should be pointed out that the same comment can be made 
about the Inquiry Development Program.) Furthermore, the project Is limited 
to only one scientific process— measur1ng--and therefore Is less comprehen- 
sive. 

Still other programs might, be considered for Inclusion, since the 
enthusiastic reception which the Initial group of government-sponsored 
projects has had among school people has led to a round of new development 
on the part of the commercial publishing Industry. At the conclusion of 
field trials, however, It seemed justifiable to restrict the package to the 
early, trend-setting programs that were funded with federal resources. To 
open the door to a whole range of the more recent programs that have come 
out of the publishing Industry would have delayed the release of the 
Information unit well beyond 1970. Plans are now underway In the Communica- 
tion Program for a "second-generation Information system" which would Include 
the capability of adding new programs to the system as they become opera- 
tionally available. It Is clear that such an updating procedure will be 
necessary to maintain the long-range effectiveness of the Information system. 
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2. It was recommended that a folder or "Review" be added In five 
copies. This folder should summarize the Information unit for Individual 
users In a group setting. Comments and observations made during the opera- 
tional field test revealed that users tended to take many notes and were 
therefore unable to follow the audiovisual elements closely. Providing them 
with the key Information and charts should help them concentrate on the 
program. Individual users also wanted a "table of contents" for the 

use of the Information unit. The rest of the materials In the "Screening 
Guide" and "Guide to the Selection of an Elementary Science Curriculum" were 
placed In a booklet for group leaders who were the only ones that needed much 
of the detail provided. 

3. A "Supplement" sheet was planned for distribution one year after 
the publication of the Information un1t--to update It and forestall the 
need for revision. 

4. It was recommended that the audiovisual briefings be reshot to 
Improve their photographic quality and to secure commercial releases from 
persons shown In the slides. 

It was assumed that all these changes would be made by the commercial 
distributor (an assumption that later proved Incorrect). 

Personnel ♦ There was no addition to the staff during this period. There 
was a turnover of one clerk-typist and one research clerk. 

Perhaps It should be noted at this point that none of the staff Involved 
In the project were science educators or had a background In science. In 
fact, with one exception, none of the staff had a public school background. 

The project found that generalists with a background In communications, 
journalism, mass media, or some related field were the most valuable. On 
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two occasions, attempts were made to involve science "experts" in the pre- 
paration of some of the materials and in both cases the results were unsatis- 
factory. This has been the experience of the staff In working on Informa- 
tion units in other subjects as well. The problem seems to be that "experts" 
frequently have biases for one particular philosophy of education; they seldom 
have an objective view that enables them to write ubout all projects with 
equal clarity. 

Time and Costs . The operational form was developed and tested between 
March, 1969 and Oune, 1969. The total costs for this stage were $47,800. The 
revision and development costs were $26,000, the field testing and data 
analysis costs were $14,500 and the overhead costs were $7,300. Of the total 
$47,800, personnel costs amounted to 56% and about $3,500 was spent on media 
work. 
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STAGES OF DEVELOPMENT 

Stage 5: The De velopment of the Release Version, the Planning for 

DIssuiffHat ion, a nd Product bis semi nation . 

Pu rpose of this stage . The Laboratory describes the phases of 
this stags In the following manner: 

Operational product revision , Includes the correction of the 
minor deficiencies discovered in the operational test. It may also 
Include those revisions judged by the potential product distributor as 
essential for adoption by schools, provided they will not In any way 
reduce the effectiveness of the product. 

D issemination planning , usually requires widely varying amounts 
of time and effort depending on the nature of the product; work on this 
stage often Is Initiated simultaneously with Stages 5 or 6; and Is 
ordinarily the Joint responsibility of the appropriate product-develop- 
ment program staff and the General Dissemination Program staff. The 
work generally involves Identification of and negotiation with an 
outside distributor for production and distribution of the final 
product; occasionally It may involve production of the final product 
by the Laboratory. This phase Is terminated after Stage 8 when 
arrangements for efficient product distribution have been completed. 

°roduct dissemination. Is an open ended period during which the 
developed products are being distributed on a large-scale basis to 
school users. The General Dissemination Program has almost complete 
responsibility for this phase, with the product-development staff 
Involved only In a periodic monitoring role to insure that the products 
are still effective In accomplishing their objectives and that they 
are generally being used In the way for which they were designed. 35 

It should be noted that at the date of this writing, the Elementary 

Science Information Unit has just begun the product dissemination 

phase. As a result, no detailed report of this activity can be made 

except to note that pre-publication sales of the Information unit 

were approximately 130 at $75 each. 




35 The Far West Laboratory, 0^ cit. 
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The chapter will be outlined In the following sections: contacts 
with publishers, development of thi dissemination plan, developing the 
final form, personnel, time and budget. 

Contacts with publishers . Under the general guidelines now used 
by the Office of Education, It Is desirable to Involve commercial 
publishers In the final dissemination of an educational product developed 
with federal support. Previously, the strategy was to make materials 
available through the "public domain." In practice, this procedure 
had prevented the release of most "project curriculum materials." Unless 
a potential publisher could be assured protection of his Investment In 
the publication and marketing of material, the risks were too great. 

Under the present poliry, copyrights can be granted for five years, 
hopefully attracting the commercial sphere to products of federally 
supported efforts and Insuring their wider distribution and use. 

Accordingly, a general announcement of the availability 
Elementary Science Information Unit and a request for proposals (RFP) was 
made through standard publishing channels. Permission to secure a copy- 
right was also granted by the Office of Education. 

During the formal waiting period for the deadline for proposals, 
three commercial publishers contacted the ' Moratory and expressed Interest 
In the product. One non -commercial it a *Uu expressed inv*re$t. Detailed 
Information was given to these prospective pitchers as provided by the 
terms of the RFP. 

At the conclusion of the formal waiting period, only the non- 
commercial publisher submitted a proposal. The probably reasons 
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for disinterest on the part of the commercial publishers (as Judged 
by the staff through Informal contacts with the publishers after the 
deadline for proposals had passed) was that (1) the Information 
unit did not have a high volume sales potential and therefore, (2) 
profits would not be satisfactory, (3) involvement by a publisher who 
either hod or might have an elementary science program would place 
him In the position of selling a competitor's product, (4) schools 
were not accustomed to paying for the type of Information contained In 
the unit and (5) salesmen didn't knpw how to market It. 

After some deliberation, the non-profit publisher's proposal 
was accepted even though It did not meet all the criteria. In particular, 
this publisher’s financial position was very weak and he had no previous 
experience In distributing a "package" of the kind Involved. Furthermore, 
his approach was to be "direct mall" since he had no field salesmen. 

The only other option the Laboratory had was to publish the 
product themselves or begin again to look for a commercial publ 1 she 
The concern for the time value of the materials ruled out anything 
that would delay publication. With great hope and a deep breath, 
negotiations began with the non-commercial publisher. 

Midway through this process, which took much longei than the 
Laboratory nad expected (fully one year prior the time original 
contact was made and negotiations were finally broken off), It 
became clear that certain conditions separated the two parties. The 
Laboratory was Interested In keeping the price at a reasonable level 
and the small cash base of the publisher necessitated the production 
of only a small number of copies, thereby preventing the cost breaks 
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customary for larger publishing rur.s. 

To keep the price to the user down and circumvent some 
difficulties that had arisen In contract negotiations, the decision 
was made that the Laboratory would create the basic Inventory of the 
units and sell them to the publisher on a cash basis. Even this 
arrangement could not be concluded however, and negotiations were 
subsequently broken off--w1th the distributor permitted to complete 
pre-publication sales he had secured by a direct mall campaign 
conducted during negotiations. 

* 

Steps were then taken to find another publisher/distributor 
who would handle the Laboratory-produced stock. By Informal contacts 
with other publishers, such an arrangement Is now being concluded. 

Developments of the dissemination plan . Before contacts with publishers 
began, a dissemination strategy was adopted. Briefly, the plan called for 
a "two step-flow" model of dissemination. That is, It was assumed that 
one of the chief avenues for spreading the use of the Information unit 
was to put It In the hands of "change agents" or "cosmopolitans" who would 
carry the unit or word of It to local adopters. This plan called for 
a marketing emphasis on regional and national figures, who were thought 
to be "Influentlals" In guiding the public school pract1ce--teachers In 
well known pre-service, science education programs} consultants who "ovake 
the rounds" at conventions; etc.} and professional organizations who hire 
out on a contract basis to assist schools In plann1ng--all this In addition 
to direct contact with th* primary target of school principals and administrators 
responsible for organizing and focusing upon local curriculum decisions. If 
feasible, this plan called for a "training" program for these "Influentlals" 
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as well as the repackaging of the information unit for use in pre-service 
programs. Because of the difficulties of securing a publisher, however, many 
of the plans have had to be curtailed. It Is still hoped they can be 
activated* 

Developing the final form . Prior to the time the formal request for 
proposals went out to publishers, It was decided to re-shoot the 
film strips In order to obtain commercial releases from the partici- 
pants shown on the films. Non -commercial , educational releases had 
been secured at the tiine of original shooting, but It was felt that 
the availability of these non-commefclal relea$es--and not the 
commercial ones--wouk H put the non-commercial bidders at an advantage 
In the bidding process. Though this work was relatively expensive, It 
was believed to be necessary; it also permitted another opportunity 
to Improve the quality of the product. The changes In the text, 

Including the addition of the report and filmstrip on COPES, as 
recommended by the Laboratory review committee, were also accomplished 
at this phase. 

When It became clear that the Laboratory would actually have to 
produce the basic stock of Information units, this work was also 
begun. It Involved selecting an overall design scheme for the 
product (an outside design firm was employed), setting the printed 
material Into print, making the master negatives and copies of the film 
strips, production of the box, and collating. This work was all 
done under contracts with outside commercial vendors. A picture of 
the final product Is shewn In figure 7. Pictures of the components 
at each of the levels within the Information unit are also shown. The 
levels and the elements within them were: 

o 
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Introductory materials: a set of instructions for users and group 

leaders in printed form; a handout (five copies) for Individual 
participants in a group-decision making activity; and a twelve-minute 
audiovisual introduction intended for large groups (or individuals). 

(See Figure 8) 

Audiovisual briefings: one briefing for each program; the brief- 

ings focus on classroom use of the programs. (See Figure 9) 

Detail program reports: one report for each program. The sections 

of the report are: (See Figure 10), 

1. Goals and Objectives 

2. Content and Materials 

3. Classroom strategies and Activities 

4. Implementation Requirements 

5. Evaluation 

6. Project History 

Personnel . As already suggested by previous text, a number of 
additional personnel were necessary to conclude this phase of the 
development. Legal advisers were used to conduct the contact 
negotiations, the Laboratory's dissemination coordinator was deeply 
involved in contacting potential publishers and suggesting patterns for 
marketing, outside vendors were placed under contract to design the 
completed form of the information unit, do the page layouts, create 
the master negatives, make copies, type set, print and bind the booklets, 
construct the box, and collate. One additional man was required during 
this period simply to coordinate all of the various details involved. 

Many of the Laboratory's senior staff were also involved, periodically, 
as decisions were made in the actual negotiating process. 
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Figure 7 

The Final Release Form of the Elementary Science Info rm ation Unit 





Figure 8 

Level I: Introductory Material s of the Final Release Form 







Figure 9 

Level II: Intermediate Audiovisual Briefings of the Final 

Release Form 





Figure 10 

Level III: The Detailed Reports of the Final Release Form 
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Time and Costs . The development of the final form and dissemination 
work to date amount to $59,100. A breakdown of this total Is as follows: 
Final form revision Including new 



Information gathering for COPES $ 47,500 

Dissemination planning 2,000 

Report preparation 8,000 

Staff training (these figures had 
not been broken out before) 1,600 



$ 69,100 

(Overhead was not computed separately at this time.) Of this figure, 
approximately 61 ' went for personnel; media costs were $10,000. Not 

t 

included in the above figure were these additional costs: $33,000 to 
create the stock of the Information units (this money should oe replaced 
as the stock Is sold) and an unspecified amount of legal fees; these have 
not been determined as yet. It should be noted that the cost', of this 
phase would have been considerably less if a satisfactory arrangement with 
a commercial publisher could have been made. We estimate the cost would 
have been about $23,000 if such an arrangement could have been made. 
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CONTINUATION OF THE INFORMATION UNIT PLAN TO OTHER UNITS 

One of the emphatic points with which this document should conclude Is 
the Idea that the systematic development of a product through the procedures 
outlined above Is an important step toward the development of a generic 
production system which, in turn, can produce ether, similar products. 

To justify the high costs of Initial development of a unit like the one 
described, It is Important to secure a 'pay off" or return of investment by 
applying the procedures, developed through this effort, to the production 
of similar products. Such was the clear intent of the strategy followed in 
developing the Elementary Science Information Unit. It was seen simply as 
a first, full scale prototype of many other Information units to come. 
Detailed records of procedures used were kept and actions were taken to 
systematically develop a technology that would replicate procedures t.o 
produce other Information units more economically . This technology is 
based, among other things, on secondary products such as manuals and 
training programs for writers and scriptwriters, detailed schemes for 
curriculum analysis and information collection and processing, and systematic 
procedures for selecting programs and involving both "expert" consultants 
from relevant fields as well as members of the target audience. 

A separate task for this effort to develop a technology for applying 
the idea of Elementary Science Information Units to other subject areas. 

This task was called "system development." Over the period of the three 
years that the work on the Science Unit was underway, the cost of this 
effort was $55,000. 
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In line with this "systems development approach" the effort behind the 
Elementary Science Information Unit was extended during 1969-70. A 
substantial savings In dollars should be noted In these further efforts 
because of the work Involved In developing the Science Information Unit. 

Finally, the most rewarding "pay off" has come while we worked with 
the first system -the Information unit system described above. As we 
gained experience and insight Into the general applicability of this system 
and recognized Its assets and shortcomings, we have reached the point where 
our developmental efforts now allow us to go beyond the system for 
developing information units to a more comprehensive system that us^s the 
best features of the first system, but adds characteristics designed to 
enhance the overall goals of the program's efforts. Such a system Is 
now under development. It is called ALERT--Alternative$ for Learning 
Through Educational Research and Technology; a detailed plan for development 
of this system Is available from the Laboratory. The document outlines 
the complete specifications of the product and is, in our opinion, unique 
in education. It has been made possible only because of long-range support 
for continuous development coupled with a rigorous systems development 
strategy. 
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